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TECHNICAL MEMORANDUM 


ESTIMATING THE COST OF MAJOR ONGOING COST PLUS 
HARDWARE DEVELOPMENT PROGRAMS 


INTRODUCTION 


The initial cost estimates for major hardware development programs at MSFC are 
by the Program Development Directorate during phases A/B. Once these programs move into phase 
C/D the cost forecasting is pnmarily done by the Program Control Office of each program. The 
cost'estimates of active then become a function of each contrac.orandtheex^ 

ence and approaches of individuals within the Program Control Offices. Therefore, the approaches 
to cost estimating vary widely from program to program. 

This report is to initiate uniform approaches to cost estimating. Three techniques are pre- 
sented: a bottom line schedule assessment technique (SAT), a detailed cost estimation (DCE) 
approach, and an intermediate cost element analysis (CEA) procedure. 


SCHEDULE ASSESSMENT TECHNIQUE (SAT) 


The SAT is a procedure to determine current and outyears funding requirements based on 
past schedule trends. The best way to determine where you are going (cost and schedule) is to oo 
at where you have been. This technique looks at historical schedule performance and forecasts t is 
performance into the future. This projected future performance is compared with the latest plan to 
provide delta costs required to complete the program. 

SAT is a bottom-line approach. It provides a uniform top-level assessment technique for all 
projects. The validity of the estimated cost is a function of how long the program has been in 
existence That is, the longer the schedule trend, the more accurate the forecast. The results ot 
SAT will usually be startling, but experience shows that projected program overruns are consistent- 
ly underestimated. 


The SAT utilizes rollover of work, new work, actual costs, and actual schedule loss to 
determine current-year and next-year cost estimates. Rollover, new work, and average histonca 
cost increases and schedule loss rates are used for forecasting outyears impacts. 


The following is a step-by-step explanation of the SAT. A calculated SAT example follows 
this section. 


NOTE: All fiscal year (FY) phasing of funds must be converted to constant year dollars. 


CURRENT FY 


• Determine roll-over from previous fiscal year 

• Determine new work 

• Determine schedule loss cost impact. 

Rollover 

1. Determine how much cost (underrun/overrun, -/ + ) was rolled over from the previous 
FY and unaccounted for in the current year plan. 

New Work 

2. Determine known content changes to be added to the current FY. Express this as 
months. Divide the dollar value by the average planned bum rate for the quarter(s) the content is 
to be added to the current plan. 

Schedule Loss Cost Impact 

3. Determine how far behind schedule the project currently is in months. 

4. Determine schedule loss rate from item No. 3 in ratio of months/month. 

5. Determine schedule remaining in current FY in months. 

Fy 6 Determme burn rate ratio of actual/plan cost expenditures to present time from beginning 

7. Determine planned bum rate for remainder of FY. 

8. Determine work remaining in months: 

Items (2 + 3 + 5) x (1+ Item 4) 

9. Determine measurable work as percentage of total for current FY. 

10. Determine cost of remaining work for current FY: 

((Item 8 - Item 5) x Items 6x7x9] + (Item 5 x Items 6x7)+ Item 1 
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NEXT YEAR 


Rollover 

11. Determine how much work from the current year will be rolled over into the next year 
(in months work). 

[current year cost ceiling - actuals - Item 10] [Item 9/actual bum rate] 

+ [Item 5] [efficiency loss] [reduced bum rate/next year bum rate] 

New Work 

12. Repeat step 2 for this FY. 

13. Determine bum rate plan for next FY. 

14. Determine measurable work as percentage of total for next FY. 

15. Determine delta months of measurable work for next year 

(Item 11 + Item 12 + next year’s planned months) x [Item 14] x (Item 4) 

+ Item 1 1 + Item 1 2 

16. Determine cost of next year’s delta measurable months 

Item 15 x Item 13 x Item 6 

17. Determine initial planned cost of next year planned months 

Planned months x Item 13 

18. Determine total cost for next year versus plan 

Item 16 + Item 17 versus Item 17 

OUTYEAR 

Rollover 

19. Determine how much measurable work from the next FY will be rolled over into 
outyears (in months). 

[next FY ceiling (months) - Item 15] . 
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New Work 


20. Repeat Item 2 for outyears. 

21. Determine total planned runout months remaining in program by program phase. 

22. Determine remaining outyear work in months 

(Items 19 + 20 + 21 fin months]) x M s 

where M s is the outyears schedule multiplier which is a function of S, and where you are in the 
program. S, is a schedule loss rate factor defined as S, = [1 + Item 4], Table 1 gives the M s as 
related to S ( - and program phase. 


Table 1 . M s as a function of S, and program phase. 


Where S^. < 1.5 



j Outyears Schedule Multiplier (MJ | 

Current Time 

PRR-PDR 

PDR-CDR 

CDR-DEL 

ATP-PRR 

1.5 

1.5 

1.7 

PRR-PDR 

s r 

1.5 

1.7 

PDR-CDR 

- 

s r 

1.7 

CDR-DEL 

- 

- 

s r 

Where 1.5 > Sj. < 

1.7 



ATP-PRR 

s r 

s r 

1.7 

PRR-PDR 

s r 

s r 

1.7 

PDR-CDR 

- 

s r 

1.7 

CDR-DEL 


- 

S f 

Where S^. >1.7 


ATP-PRR 

s r 

1.7 

1.7 

PRR-PDR 

s t 

s r 

1.7 

PDR-CDR 

- 

s f 

S f 

CDR-DEL 

- 


S f 


-3. Spread Item 22 in proportion to the initial baseline outyears planning from the nearest 
appropriate major milestone. For example, if you are at critical design review (CDR) use the 
remaining baseline plan months to ratio the increased months. For example, a planned milestone 
which was hallway between CDR and delivery will remain halfway between. 

24. Determine outyears cost 


|(Total program planned cost)XC ( | - actuals 

where C, is multiplier depending upon where you are in the program and the bum rate ratio (B 
(Item 6). (See table 2.) 





Table 2. Q as a function of Br and program phase. 


When Br < 1.6 


Current Phase 

Out -years Cost Multiplier (Cj.) 

atp-prr 

1.6 

PRR-PDR 

1.5 

PDR-CDR 

1.4 

CDR-DEL 

1.1 


Where Br > 1.6, use Br as 
Where Br < 1, use Br as 


25. Spread outyears cost at same proportion as initial baseline plan. For example, if you are 
in the pre-CDR phase and the initial CDR expenditures plan called for 50 percent of cost, then 
spread the forecast costs similarly. 

26. Convert the constant year dollars to real year dollars. 


SCHEDULE ASSESSMENT EXAMPLE 

Assumptions m FY2 FY3 FY4 FY5 

5-Year Program Authority to Proceed (ATP) 

PRR PDR CDR MAN DEL 

Current status is 8 months into FY2. 

Plan bum rate FY1 is $4M/M; FY2 $5M/M; FY3 $6M/M; FY4 S5M/M; FY5 $3M/M. 
Rollover from FY1 is $4M. 

Known content changes $1.25M in FY2, $3M in FY3, $1.25M in FY4. 

Project is 2 months behind schedule (from ATP) not including rollover. 


Actual bum rate for FY1 has been same as plan. 

Actual bum rate for FY2 has been S5.5M/M. 

Measurable work is 75 percent of total (for FY2 and FY3). 
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Preliminary design review (PDR) is mid-FY2. 

CDR is mid-FY3. 

Manufacture start is mid-FY4. Manufacturing/assembly/testing are not excessively complex. 
Delivery (DEL) is end FY5. 

Assume FY2 and 3 are capped at S60M and $72M. 

All dollars are constant FYI dollars. 

1. Unaccounted cost rollover from FYI =0. 

2. Known content change for FY2 is $1.25M, which gives $1.25M/$5M/M = 0 25 
months. 

3. Program is 2 months behind schedule (given). 

4. Schedule loss rate is 2 months/2() months = 0.1 month/month. 

5. Remaining schedule in FY2 is 4 months (given). 

6. Burn rate ratio for FY2 is S5.5M/M actual/$5M/M = 1.1. 

7. Planned burn rate for remainder of FY2 is $5M/M (given). 

8. Work remaining is; 

|0.25 (Item 2> + 2 (Item 3) + 4 (Item 5)J x ( 1 +0.1) (Item 4) 

(0.25 + 2 + 4) (1 +0.1) 

6.8 months 

9. Measurable work is 0.75 (given). 

10. Cost of remaining work for current FY is: 

|6.8 (Item 8)-4 (Item 5)] (1.1 (Item 6) x 5 (Item 7) x 
0.75 (Item 9)| + 4 (Item 5) 1 1 . 1 (Item 6) x 5 (Item 7)| + (> (Item 1) 

which is: 

|6.8 — 4|[l.l x 5 x 0.75 1 + 4| 1 . 1 x5| = 11.6 + 22 = $33. 6M 
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NEXT YEAR 


Rollover 

1 1 Assume funding cap for FY2 is S5M/MX 12 = $60M. Actuals of SS.SWMXSM = 
$44M plus runout of $36.6M puts a funding problem of *17*M. We only have ^MJor^ 
remainder of FY2 so we have to reduce the FY2 bum rate. Therefore, we p $ • 

next year which is at the FY2 actual burn rate and 75 percent measurable gives 2.4 months. Note, 
however, that by reducing the FY2 bum rate by 27 percent over the remj.ng 4 months we 
would reduce efficiency as well. Assume we had a maximum efficiency of 75 percen • ' hen ’ 
would achieve 3 months of work at $4M/M and push 1 more month into FY3. This 1 mondl a 
S4M/M versus S6M/M would equate to 0.67 FY month. Therefore, we push a total of 3 months 

into FY3. 


New Work 

12. Known content changes (given): 

PY3 = $3M @ $6M/Month = 0.5 Months 

13. $6M/M bum rate for FY3 (given). 

14. Measurable work for FY3 is 0.75 (given). 

15. Delta months of measurable work for FY3: 

[3 months (Item 11) + 0.5 months (Item 12)+ 12 (FY3 months) x 
(0.75) (Item 14)]x0.1 (Item 4) + 3 (Item ll) + 0.5 (Item 12) 

= 4.7 measurable months 

16. Cost of next year’s delta measurable months. 

4.7 (Item 15) x 6 (Item 13) x 1.1 (Item 6) = $31M 

17. Planned cost of next year’s planned months: 

12 (planned months) x 6 (Item 13) = $72M 

18. Total cost versus plan 

$31M (Item 16) + $72M (Item 17) 

$103M versus $72M 
Delta of $31M 
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OUTYEARS 


Rollover 


19. Measurable work from FY3 to be rolled over: 

4.7 months (Item 15) since FY3 is capped at $72M (12 months at $6M/M) 

New Work 


20. Known content change for FY4 is $1.25M (given). 

$1 .25M/5M/M = 0.25 months 

21. Total planned outyears runout months is 24 (FY4 and 5). 

22. Total remaining outyear work in months: 


[4.7 (item 19) + 0.25 (Item 20) + 24 (Item 21)] M s 


where M s = 1.7, because S f = 1.1 < 
CDR-DEL, which gives = 49 months. 


1 .5 and program phase is PDR-CDR and outyears are 


3. For the outyears spread Item 22, the new months required to complete the project on the 
same percentage as the initial plan. From the 8th month of FY2 percentage of time was 22/40 to 
manufacturing start, 40/40 to delivery or 55 percent manufacturing, 100 percent to delivery. There- 
ore, time to manufacturing start is 55 percent of 65 (remaining time in FY2 + FY3+ Item 22) 

100 percent of 65 to delivery. CDR slips 5 months (Item 15). 



WAS 

IS 

CDR 

Mid-FY3 

End of FY3 

Manufacturing Start 

Mid-FY4 

9th Month of FY5 

Delivery 

End FY5 

1st Month of FY8 


24. Outyears cost: 


Total planned program cost x C t - actuals; C, = 1 .4 where 1 < B r < 1 .6, B r = 1 
FYI+FY2 + FY3+ Rollover (Item 18) + FY4 + FY5- actuals (FY1 + FY2 + FY3) 
[($ 1 80M + $3 1 M + $ 1 2M x $5M/M + $ 1 2M x $3M/M) x 1 .4] - ($ ] 80M) 

[($307) x 1.4] — $180M = $250M 
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25. Spread costs in proportion to baseline: 


PLAN 
FY1 $48M 
FY2 $60M 
FY3 $72M 
FY4 $60M 




FY5 $36M 


% 

$M 

CDR 

52 

144 

Manuf. St. 

lb 

210 

DEL 

100 

276 


FORECAST 


ADJUSTED 

$48M Actual 
$60M Ceiling 
$72M Ceiling 

1 $250M 

FY8 

% 

$M 

% 

$M 

52 

223 

44 

180 

76 

327 

lb 

327 

100 

430 

100 

430 


Since FY1-FY3 were capped at their ceilings (and for the most part actuals) we could not cost 
$223M at CDR. but instead $I80M. However, the outyears funding P^e was ^eveioped * ^ 
had costed $180M at CDR and then the difference between $223M-$180M - $43M was sp 
over the timeframe between CDR and manufacturing start. This funding spread was developed 
direct proportion to the baseline plan considering months between milestones and varying 
rates. This resulted in the following forecasted funding profile. 



FY1 

FY2 

FY3 

FY4 

FY5 

FY6 FY7 

FY8 

$FY1(M) 

48 

60 

72 

87 

66 

45 34 

6 

26. 

Convert 

to real year dollars (at 

5 % escalation) 


$FY(M) 

48 

63 

79 

100 

79 

56 44 

8 
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DISCUSSION 


Using this example's S, of I . I , B r of I . I (both of which are relatively low), and program 
phase ol between PDR and CDR (which dictates an M s of 1.7 and a C f of 1 .4) gives a proiected 
tota program slip of 1.43 which compares with NASA's recent experience on major pregrams of 
I.M) and a cost overrun ol 1.56 which compares with recent experience of 1.6. 


This example was worked based on nominal experience. That is, usually the operating year 
and the next year have funding constraints and the outyears are flexible. For academic purposes 
we could work two other opposite scenarios; 


1. All funding of FY1-FY5 is capped and therefore all schedule loss rates and increased 

costs would be pushed beyond FY5. This approach gives a resultant “plan” which is totally 
unrealistic. J 


2. We could also determine to complete the program within the calendar 5 years. This 
would necessitate a significant increase in bum rate to accommodate the increased measurable work 
due to the schedule loss rate. However, if the increased funding were available (highly unlikely) 

and the contractor could accommodate such an increase, this would be the least expensive and 
preferred approach. 

II we chose to ignore other program's historical data and the use of M s and C f , but rather 
used only the S, and B r of this particular program, we would have forecast a projected total 
program slip of 1.15 and a cost overrun of 1.17. Essentially, using recent program cost/schedule 
performance can give quite accurate forecasting for the current and next FY. But, without the use 
of historically derived cost and schedule multipliers (M s and C f ), the outyears forecasts would only 
project continuing the same past performance. This, in most cases, would be significantly in error 
because the contractor performance varies with the program phases. For this example, without the 
, s a " d ^ would have forecasted delivery 16 months (23%) earlier and cost at $107M 
, V ess than SAT Therefore, in forecasting the outyears, one should use these derived 
multipliers which more accurately reflect recent NASA programmatic history. If one chooses to use 
other multipliers, then a strong substantive rationale must be developed that is devoid of wishful 
thinking, unbridled optimism, and fear of the consequences of “truth in forecasting.” 

NOTE: The outyears multipliers (cost and schedule) were determined using recent NASA program 
history. See SAT Supporting Rationale. V S 


SAT SUPPORTING RATIONALE 


For the SAT to ^ accurate, it is imperative that schedule performance as related to the 
schedule baseline be accurate. However, this is mandatory in any program that is managed properly 
regardless of the use of SAT. & K F 3 

The recognition that programs slip schedule is nothing new. Also, the recognition that 
schedule is money is nothing new. However, very seldom is past schedule performance of an on- 
going program used to forecast future schedule and cost performance. Generally, the contractor will 
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come in with an estimate-to-complete (ETC) ignoring past schedule performance and basing the 
forecast on optimism. 

In early 1964, a NASA task team researched and studied the actual schedule history of 312 
individual items of flight and ground hardware and determined a slip factor for each. The average 
slip factor was 1.71,' that is, it took 70 percent longer to complete the task than planned^ This 
same study showed that schedule slippage accelerated after approximately 65 percent of the original 
baseline schedule had transpired. These data were based on NASA programs when these progra 

and NASA were new. 

A more contemporary evaluation of schedule loss is provided in Augustine’s Laws where 
Norman Augustine used congressional data that illustrated that most all mi, ; t 7 s f t ^ s S .^; e nitial 
delivered an average of 1.33 times their baseline schedule (at an average of 1.52 times the 

cost). 

The most recently completed major hardware programs at MSFC were surveyed to de- 
mine the cost overrun and schedule loss rate factors as a function of program phase. These fa 
are listed in tables 4 and 5 in the appendix. For example, the SSME DDT&E project yielded a 
cost factor of 1.5 and a schedule factor ol 1.4. 

It should be noted that the SAT factors are based on very limited data. These factors could 
significantly change if the initial optimistic schedules and cost of new programs become more 
pessimistic^ realistic) in the future. Also, these factors would be refined by inclusion of more 

historical data. 


For a 


detailed discussion of the limited data and its interpretation, see the appendix. 


DETAILED COST ESTIMATION 


This section contains a compilation of questions and data that should be addressed in 
detailed forecasting of program runout costs. All of these data are not of the ^reanvemagm- 
tude and some data are more applicable deending upon the individual program. This informatio 

presented topically as follows: 


Phase A/B 
Budget Impacts 
Contractors 
Integrated Plans 
Schedules 
Manpower 
Cost Reporting 


'Letter No. CP10-75-219, from CP01 to DA01, dated May 2, 1975 
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Performance Measurement System (PMS) 

Design/Manufacturing/Test 

Contract Changes 

Changes 

Design Reviews 

Other 

Phase A/B 

• Was there an adequate phase A/B? 

• Were there major changes from A/B to C/D? 

• Were there significant scope/funding changes in negotiations? 

• Was there a buy-in? 

• Were there requirements deletions with associated excessive funding deletions? 

Budget Impacts 

• Has there been significant FY budget rephasing without a bottoms-up ETC? 

• Were budget cut impacts assessed by the subcontractors or just estimated by the prime? 

• Has the prime signed up to lower funding levels without significantly reducing scope? 

Contractors (Prime and Subs) 

• Have there been comparable programs or experience? 

• What is the experience of the people on the program (from managers to manufacturers)? 

• What is the union/management relationship? 

• What is the management approach? 

• What is the business base? How much is Federal? How much is NASA? MSFC? 

• Are the critical people 1 (X) percent dedicted to the program or shared with other 
programs? 
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• Do the managers come off the program too early or too late? 

• Does the prime have an adequate cost/schedule control system? 

Integrated Plans 

• Is there a detailed integrated cost/schedule/manpower/vendor plan? 

• What is the basis of this integrated plan? 

. What is the spread of technical and nontechnical manpower to the major program phases? 

• Are these ratios changing with the program progress? 

• What do these trends forecast ? 

• What is spread of time. cost, and vendors in relationship to the major milestones? 

• Do they correlate with each other - with the manpower spread? 

Schedules 

• Do detailed baseline schedules exist? Is there an adequate schedule configuration control 
system? Is it implemented in accordance with proper procedures? 

• Are the schedules integrated with cost/vendors? 

• Are there logic networks? 

• What level do the schedules breakdown to? 

• Are the schedules “user-friendly?” 

• Were the schedules developed by the performing work breakdown structure (WBS) 
managers? 

• Do the inhouse technical personnel agree with the schedules? 

• Are the schedules used and statused daily? 

• Is there a proliferation of informal schedules? 

• Is there adequate critical path definition? 

• Are these resources/facilities included in budget to cover “slack” times? 


• Is the schedule loss rate being tracked? 

• Are there real workarounds? 

• What about costs on shifts required for workarounds? 

• Is there a milestone tracking system? 

• Are the milestones weighted? 

• Are the significant milestones lost in the milestone count? 

• What are the schedule trends? 


Manpower (Primes and Subs) 

• is the manpower profile consistent with schedules and tasks? 

Often the cost and manpower profile just reflect the FY funding constraints and not real 
planning. 

• What is the allocation of manpower by discipline: Engineering, manufacturing, etc.? 

• What is the technical to nontechnical manpower ratio? You will often find that as scope 
slips out due to budget constraints that the nontechnical manpower increases percentage 
wise. This could put one into the position of appparently having adequate resources to do 
the tasks (i.e., cost and manpower), but there is in reality insufficient technical manpower 
to do the technical job. 

• What are the technical/nontechnical manpower plans to Preliminary Requirements Review 
(PRR), PDR, CDR, delivery, etc ? Is there visibility of this for each change order? Often 
the manpower to the near term milestone will increase, but the total manpower will 

remain the same. This obscures a reduction in planned manpower to achieve the outyears 
milestones. 

• What are the manpower trends overall and per subsystem? 

• What are the plans to accommodate roll-off of manpower? Is there an adequate business 
base to allow this? 


• What are the manpower levels for sustaining engineering? 

• Are there sharp breaks (curves, trends) in the manpower plan? What is the rationale to 
substantiate this? 


• Is there adequate systems engineering manpower? What is the ratio of systems engineering 
manpower to other disciplines? Other programs? History? 
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Cost Reporting 

• What are the planned bum rates? 

• Are these rates consistent with planned work? 

• What are the actual bum rates? 

• When work is falling behind, can the contractor increase bum rates to catch up? 

• Is the contractor using the salary level of employees bid and negotiated? Often more 
expensive salaried people are on the program versus what is bid. This expense adds up 
over several hundred people over a few years. 

• On the 533, are the overhead rates “predicted” rates or actuals? What is the history of 
overhead rate adjustments? What are the likely circumstances and times to increase rates? 
Rates always go up. The best thing to do is anticipate this and plan for it. 

• What is the history on general and administrative (G&A) (costs)? What are the drivers 
and the likelihood of change? 

• What is the validity of the contractor’s estimate to completion (ETC)? Was this a delta 
ETC? Did the contractor management/project control office just “pass through a compila- 
tion of costs without a thorough review? Were the outyears assessed? 

• What has been the experience of the contractor s forecast on this program? Other 
programs? Is he consistently low? 


Performance Measurement System (PMS) 

• Is there an adequate PMS? Is it being fully utilized? Most often, the PMS is a hoop 
the contractor jumps through and ignores its potential. Also, the Program Offices view 
PMS as a nuisance. The PMS and the schedules must be used by management and the 
project engineers for managing the program or else they become a useless expensive 
facade. 

• What is the relationship between level of effort (LOE) and measurable manpower. LOE 
should be kept as low as practical. Often a contractor will front load a PMS with signifi- 
cant LOE thereby obscuring the real performance measurements. 

• What is the method for taking earned value? It should reflect actual work accomplishment 
versus resources expended. Many times a prime will use “bookings” as a method for tak- 
ing earned value on a subcontract. This is a worthless technique. 

• What are the PMS trends? No matter what one does, the PMS trends are most often 
understated by a value of 2 or 3. However, the trends are usually accurate. 
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• One ought to have an inhouse automatic technique for calculating and displaying PMS 
trends at the lowest level reported. This technique would then forecast future performance 
based on continuation of past performance. It can also calculate “improved performance” 
required to get back on plan. Critical path areas should be particularly monitored. 


Design/Manufacturing Test 

• Too many “to be determineds” (TBD’s) in equipment (EQ) specs, too many changes in 
EQ specs, and too late definition/incorporation of EQ specs spell increased costs. 

• Drawing release lists by project should be planned and actuals tracked. Engineering 
change orders (ECO’s) should also be tracked. Trends and forecasts can be made using 
this. 

• Technical issues/problems not expeditiously resolved cause serious cost trouble. 

• Are there adequate manufacturing and deliverable spares? There must be visibility and 
tracking between the two. 

• Is there more than one material supplier? Are there any high risk materials that a contin- 
gency source should be considered and potentially funded? 

• What tools and handling equipment are required? Are these defined? Are they scheduled 
appropriately? Are the schedules realistic? Large programs have been known to come to a 
grinding halt because these items were overlooked or casually thought about. 

• What are the required logistics for movement through this phase? Are the facilities 
adequate? Are the facilities old and have not been used in a long time? Are the facilities 
brand new? Are they automated? Are they complex? Are the facilities available? Are 
there any other programs that plan to use the facilities? Who has priority? How is priority 
determined? Any of these items can seriously affect schedule and therefore cost. 

• What is the testing schedule? How many shifts and how many people? What is the slack 
time? It there is a second and third shift planned, will there be enough qualified people 
available for these shifts? What are the critical skills required? What is the experience 
base on manufacturing, testing? What was required on similar programs in the past? What 
was required before with this contractor? 

• What is the schedule for electronic, electrical, electromechanical (EEE) parts? Is it 
adequate? What is the experience of the vendors? What about obsolescence ? Are there 
single source vendors? Can you get a second source? What are your plans if you receive 
an inadequate lot ? 
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Contract Changes 


A system must be established where both the Program Office and the contractor understand 
the scope, cost, schedule, and ground rules of changes. Too often there is inadequate communica- 
tion before changes are issued and the changes come at a price higher than anticipated. 

Changes are often not fully technically integrated from a systems aspect which leads to 
higher cost. 

It is standard operating procedure for changes to be issued and months and even years to 
pass before these changes are negotiated. In addition to poor management, this often reflects 
unanticipated cost increases due to rates, and the contractor being “smarter; and there is precious 
little negotiating room because the contractor has a bag full of actuals. 


Changes 

Are there funds set aside for anticipated changes from design reviews? Is there schedule 
slack anticipated to handle these changes? 


What is the rate and size of changes not related to major reviews? This can indicate an 
inadequate phase B with resulting potential problems. 


Design Reviews 

Are there funds set aside to accommodate design reviews? Is the schedule adequate? Often 
the planning for design reviews will be as if there will be no schedule or cost impact from changes 
or time spent in review. A review may be planned as if the contractor can keep on working at a 
high rate whereas the contractor must really wait for NASA review and direction. This can be a 
very costly underplan in schedule. 


Other 


What are the inhouse costs? What is the trend on these? Is there adequate technical 
involvement to prohibit late changes? Is there adequate manpower planned for reviews? Have all 
the appropriate inhouse approval authorities agreed to these plans? What inhouse equipment, facili- 
ties, tools, are required? Do you have funds set aside to “contribute” your fair share in assuring 
continued inhouse expertise, equipment, and facilities? Can you allow a little reserve in the 
inhouse line? 

Have the other involved Centers been brought in early enough to be a part of defining 
requirements, etc., to avoid late changes? A poor working/communication relationship can be a 
source of unexpected increased costs. 
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What are the known risks/threats/growth/new content of the program? Are there funds to 
cover these? Are these funds available in the years these items are anticipated? Reserve funds in 
the outyears for nearyear changes are like runway behind an airplane — it does not do much good 

Test the logic, rationale, trends behind other contractor cost elements, such as ODC, 
computer time, travel, supplies, etc. 


COST ELEMENT ASSESSMENT (CEA) 


If one is not comfortable with the SAT, or knowledgeable enough or time-limited such that 
a detailed cost estimation (DCE) cannot be done, then the traditional assessment by using cost 
element breakdown is done. This CEA is simply an intermediate level assessment between the SAT 
and the DCE. It also can serve as a confidence level check. As a matter of fact, it would be best 
to independently do an SAT, CEA, and DCE to see how closely they converge. 

For a CEA, one looks primarily at the major cost drivers which can vary from program to 
program. Most often the major cost drivers are prime manpower and subcontractor cost. In addition 
to these, other items are assessed such as materials, other direct costs (ODC), other costs (includ- 
ing o/h, G&E, fee), issues, and bum rate. 


Manpower 

Use the same questions as listed under the DCE and determine: manpower trends, 
manpower loading versus major milestones, manpower allocation of technical versus nontechnical. 
Look at manpower breakdown by project. Graph these manpower allocations and compare the 
spread with the schedule. Look at the major work content items and compare the content versus 
manpower allocation. Assuming you have an ongoing project, the best indicator of future perform- 
ance is past performance. If the contractor is using more manpower to do the job, or as often the 
case, using the total planned manpower but not getting the total job done, you can project future 
performance based on this past performance. 

The last thing a prime will do is offload his manpower. Budget reductions will usually be 
passed on to subs not under contract and the prime manpower kept up. If the contractor is forced to 
reduce manpower, he usually reduces the technical manpower which is required to get the job done. 
Also, it is difficult to bring manpower down after CDR, especially if there is a low business base. 

Break the manpower plan down by work breakdown structure (WBS) or discipline and make 
future projections based on past performance by WBS or discipline and sum the total. Assume that 
the outyears manpower (for manufacturing, etc.) plans are underestimated at the same level as 
current performance. 
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Subcontractors 


Subcontracts are often awarded late so as to keep prime funds available for inhouse 
manpower. Sub costs can be notoriously underestimated. One should look at each major sub and 
determine the basis of estimate (e.g., a prime’s rough order of magnitude (ROM), or sub s 
bottoms-up). Also determine sub’s performance if the contract has been awarded. Look at the sub s 
major cost contributors, schedule, history, etc. Ask yourself the pertinent questions listed under the 

DCE. 


Using this data and past performance, project future performance on each sub and sum the 

total. 


Materials 

This is usually simple. Look at the major cost drivers in the material category. Prepare the 
cost spread with funding availability, need dates, vendors, second sources, etc. Look for trouble 
spots such as unique forgings done in only one place or exotic or long lead materials. If historical 
data are available use them. Determine who needs what and when and assure proper planning. 

Estimate materials cost by major materials and potential problems as listed above. Add in 
the remaining materials costs with a percentage increase the same as the percent you increased the 
major material items. 


ODC 


This approach is similar to materials. Look at the major cost drivers in this category. Test 
each for reasonableness. Using historical performance and the results of your reasonableness test, 
forecast future costs. 


Other Costs 

Overhead, G&A, and fee should be addressed using questions stated in the DC A. Fee is 
often planned at 100 percent and performance is rarely at this level. This could leave a small 
source of funds. Overhead and G&A adjustments will blindside you unless you anticipate it. An 
unexpected five percent increase in G&A can devastate the program. Also, on the 533, the contra 
ctor often reports provisional (planned) rates and not actuals. 

Look at the contractor's historical performance; look at the business base. Determine the 
possibility of and timing of increases and forecast appropriately. 


Issues 


An analyst attempting to forecast future costs can get into trouble just using cost data and 
numbers. One must look at other major potential threats such as major technical or programmatic 
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issues. What are the major problems in the program? What is the likelihood of a budget impact in 
resolving these issues. Are the resources (manpower, funding, schedule, subs, etc.) adequate or 
impacted? 

After this assessment is done, it is best to adjust the bottom line costs of the previously 
forecasted cost elements. For example, it may be necessary to increase the manpower estimate 
because of a known inhouse problem that can only be solved by more manpower. 


Burn Rate 


After all of the above items are estimated and summed with threats and reserves, a bum 
rate analysis should be performed. That is, compare the actual bum rates of total cost, manpower, 
and subs with the planned bum rates and the forecasted bum rates. 

Often you will find that a contractor simply cannot increase the bum rate in time to accom- 
plish the forecast. For example, if in using past performance you forecast a significant bum rate 
increase in manpower to keep up with the schedule, often the contractor cannot hire and train 
people fast enough to accomplish this; or, the FY funding constraints would prohibit such an 
increase. 

Therefore, the total bottom line forecast will have to be tweaked in accordance with 
required versus affordable bum rates and schedule. The schedule often will have to be slipped 
some, resulting in rate increases. 


WARNING SIGNALS 

If the program or contractor demonstrates several of the following characteristics, anticipate 
a cost overrun. 

• Inadequate Phase B 

• Significant Scope Changes at Negotiations 

• Buy In 

• Hardware Poor Program 

• Significant Technical Challenges 

• Major Design Changes 

• Major Budget Reductions/Rephasings 

• Bow Wave of Deferred Work 

• Parallel Development Interface With Other Programs 

• Contractor/Subcontractor Inexperience 

• Contractor Organizational Confusion/Shuffling 
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• Contractor Nonresponsiveness 

• Contractor Near Term Focus Mentality 

• Little Emphasis on Schedule Control 

• No Integrated Schedule/Logic Networks 

• No Integrated Schedule/Cost 

• No Critical Paths 

• Significant use of workaround schedules 

• Inadequate Systems Engineering 

• Adverse Cost and Schedule Trends 

• Inadequate Capability to Validate Baseline 

• Lack of Performance Assessment Visibility 

• Reluctance to implement appropriate PMS 

• Significant TBD’s/Number of Changes in EQ Specs 

NOTE: These should be a part of every program operating plan (POP) review. 

RECOMMENDED FURTHER RESEARCH 


Historical data are most often used to initially forecast an ETC at ATP. but rarely used to 
apply to ongoing programs. SAT and other techniques should be developed to provide uniform 
comparative estimates for ongoing C/D programs. 

More contemporary NASA programs should be assessed to more accurately determine 
schedule loss and the cost increase rate factors. These factors should be studied to determine if 
there is any correlation with other variables. Perhaps the rates are significantly different for manned 
and unmanned programs. Perhaps they vary as a function of complexity, dollar value, weight, con- 
tractor, etc. Particularly, since the Challenger accident, the rates and factors may have significantly 
increased. Considering this, one may wish to increase both S t and C t until the impact of the 
Challenger adjustments are determined. 

A simplified, summary historical data base should be developed reflecting this approach, 
appropriate data and current Program Development data and techniques. This data base should be 
readily available to Program Managers and Program Control Chiefs. Instructions to use this data 
base should be simplified and automated. 


The sections on Detailed Cost Estimation and Cost Element Assessment should be reviewed 
by program personnel - managers, chiefs, cost and schedule personnel. These sections would then 
be expanded and revised appropriately. A new section should be developed to include as many 
pertinent historical examples as possible. Charts, graphs, and summary data readily recognizable 
and useful should be included. The historical examples would illustrate and reflect the findings 
presented in the SAT, DCE, and CEA sections. 
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Automation ot the SAT should be developed and implemented across all programs. 


Uniform cost assessment approaches for other type activities should be developed including 
small payloads, operations activities, and production programs. 

A system should be so devised to keep track of schedule loss and cost increase rates for 
ongoing programs. This is not done now and real time or recent historical analysis is like pulling 
teeth with no tools or teeth. This would keep a current data base for revising rates for forecasting 
as well as give continuous "readout” to management on whether NASA programs are trending in 
positive or negative directions. 
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APPENDIX 


This section includes a listing of the limited data available and a discussion and derivation 
of the schedule and cost factors and multipliers as listed in tables 1 and 2. 


Table 3 is a list of schedule and cost performance data for the MSFC space shuttle main 
engine (SSME), solid rocket booster (SRB), and external tank (ET) design, development test, and 
engineering (DDT&E) projects. These data were analyzed, compared with the Holbum, Augustine 
and NASA 1 experience, and used to determine the schedule loss rate factors and multipliers or t e 
SAT. Tables 4 and 5 summarize these data. 


As you can see, the SSME project demonstrated a project slip from an ATP delivery of 
July 1977 to an actual of August 1979, resulting in a slip factor of 1.4. Further analyses demon- 
strates that the slip factor between planned ATP-CDR and actual ATP-CDR was 1.35. The planned 
(ATP) CDR-DEL time increased by 1.48. This demonstrates the observation by the Earl H.lburn 
team that schedule slippage accelerates toward the end of the program. Also, please note that if the 
planned delivery had been slipped by approximately 1 .4 (and the planned CDR) at the time o t e 
PDR, delivery would have been accurately forecast 7 years in advance. 


Since the SSME was the “leading” project, its slip factors are used in lieu ot averaging with 
the ET slip factor (2.12) and the SRB (1.85). The ET and SRB project deliveries were timed with 
the SSME. If the SSME project slipped significantly, the ET and SRB schedules were adjusted 
accordingly. (All the deliveries were within 4 months of each other.) However, a few observations 
can be made. Both the ET and the SRB projects show accelerated schedule slippage after CDR. 
Also, if at the ET ATP, the PDR would have been forecast using the SSME slip factor ot 1 .4, it 
would have been precisely correct. Using the 1 .4 at an estimated SRB project start of December 
1972 would have placed the PDR within 1 month of where it actually occurred and forecast with 
over 90 percent accuracy the CDR. In summary, all three major projects showed consistency with 
a 1 .4 slip factor. However, the SRB and ET schedule slippage after CDR exceeds 1 .4 significant- 
ly It is supposed that this slippage was incurred to minimize real year expenditures while main- 
taining required delivery dates consistent with SSME and orbiter needs. Using the slip factor tor 
SSME and comparing it with the Hilburn, Augustine, and Agency slip factors results in the recom- 
mended SAT S, of 1.6 spread by program phase in proportion to the SSME experience. 


In analyzing the cost, the following total cost factors from ATP are observed: ET (0.7); 
SRB (0.88); SSME (1.5); Augustine (1.5), NASA (70’s) of 1.4, and NASA (80's) ot 1.6. The 
SRB began as an inhouse program ana progressed contractually and significant changes were rna e 
Obviously, significant program assumptions/directions were made early on in the ET program. The 
SRB and ET programs showed no correlation until PDR when forecasts behaved in similar patterns 
to the SSME project. Therefore, an analysis of these data results in the recommended cost 
multiplier per program phase for outyears forecasting as listed in table 4. 

Tables 6 and 7 list the forecast per program phase (major milestone) for the SSME and ET 
programs. The POP’s were reviewed to determine escalation rates and reduce all forecasts to con- 
stant dollars (71$) for comparison. Similar data for SRB are available. 


'Data from NASA Advanced Project Management Course. 
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Table 3. Schedule and cost performance data. 
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Table 4. Schedule slippage factor Sf 
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Table 6. SSME DDT&E costs 
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